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Foreword 


This  final  report  summarizes  the  results  of  a  three  year  research  effort 
undertaken  by  the  School  of  Electrical  Engineering  and  the  School  of  Aeronautical 
and  Astronautical  Engineering  at  Purdue  University.  The  primary  purpose  of 
this  work  has  been  to  investigate  the  various  aspects  of  target  extraction  and 
recognition  in  video  data,  and  tracking  of  these  targets  in  sequential  imagery.  As 
indicated  in  detail  in  Section  III,  four  Ph.D  theses  and  three  M.S.  theses  resulted 
from  this  research,  as  well  as  30  additional  publications.  The  authors  are 
indebted  to  the  12  graduate  students  listed  in  Section  III  who  provided  the 
programming  and  many  of  the  good  ideas  presented  in  these  publications. 

It  is  gratifying  to  know  that  the  results  of  this  research  will  not  culminate  in  only 
a  report.  This  research  has  been  directly  supported  by  the  army  laboratories  and 
many  trips  have  been  made  between  Purdue  University  and  Picatinny  Arsenal  to 
install  and  test  software  and  to  make  comparisons  with  existing  algorithms. 
Additional  contracts  have  been  spawned  by  this  basic  research  contract  which 
have  resulted  in  more  detailed  and  specific  tests  and  implementations. 

The  authors  would  like  to  thank  Dr.  Frank  Kuhl  of  the  U.  S.  Army  ARDEC 
and  Mr.  William  Cadwallender  also  of  ARDEC  for  their  interest,  support,  and 
research  contributions  to  this  project.  We  also  express  our  appreciation  to  Dr. 
William  Sander  of  the  U.  S.  Army  Research  Office  for  his  support  and  assistance. 
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L  Statement  of  the  Problem  Studied 


The  research  problems  studied  during  this  research  contract  emphasized  certain 
sections  of  the  original  proposal  due  to  funding  limitations.  The  fundamental 
research  problem  addressed  was  that  of  extracting  useful  target  information  from 
video  imagery  and  providing  estimates  of  the  target’s  future  position.  Video 
processing  methods  are  used  to  provide  estimates  of  an  aircraft's  orientation  in 
space.  The  three  orientation  angles  can  be  used  in  a  target  tracker  to  improve 
tracking  performance.  Advanced  trackers  of  this  type  utilize  a  mathematical 
model  of  the  force  system  on  the  aircraft  that  is  a  function  of  vehicle  attitude  and 
velocity.  Target  aircraft  do  not  fly  in  straight  lines  during  tactical  situations. 
Rather,  they  will  accelerate  in  a  violent,  time-varying  fashion  (i.e.,  jink)  to  avoid 
being  shot  down.  The  advantage  of  jinking  is  that  the  future  trajectory  of  the 
vehicle  becomes  more  difficult  to  predict,  making  computation  of  the  gun  lead 
angle  substantially  less  accurate.  Lead  times  of  1  to  4  seconds  are  typical  and 
during  this  period  an  aircraft  can  maneuver  away  from  tracker  gun  fire. 
Orientation  was  found  to  be  useful  in  tracking  because  when  a  vehicle  is  going  to 
turn,  the  first  thing  that  happens  is  the  orientation  changes.  This  has  the  effect  of 
tilting  the  lift  or  thrust  vector  that  acts  on  the  vehicle  and  the  vehicle  trajectory 
turns  under  the  influence  of  this  new  force  system.  Because  orientation  always 
leads  trajectory  changes,  orientation  is  useful  in  determining  where  the  vehicle  is 
going  in  the  future;  i.e.,  in  predicting  future  trajectories. 

Thus  two  primary  research  areas  were  addressed:  image  processing  necessary  to 
derive  recognition  and  orientation  in  video  data,  and  tracking  algorithms  which 
utilize  the  orientation  information.  Work  in  image  processing  was  based  on 
previous  efforts  which  had  been  developed  for  recognition  of  low  noise  complete 
contours  in  image  data.  Research  in  this  project  has  emphasized  images  which 
contain  more  difficult  imagery  either  due  to  noise,  resolution,  or  lack  of  contrast 
with  the  background.  The  areas  of  segmentation,  partial  shape  recognition,  and 
recognition  of  range  map  data  have  been  studied.  Other  related  image  processing 
research  topics  were  addressed  during  the  course  of  this  project.  These  include 
image  compression,  precision  measurements  in  images,  and  the  use  of  image 
morphology  processing.  One  general  purpose  of  this  project  has  been  to  develop 
and  provide  software  for  research  and  development  in  tactical  target  recognition. 
The  software  has  been  developed  on  a  Sun  4/280  system  but  has  been  modified  to 
run  on  various  Unix  systems. 

Two  new  classes  of  target  trackers  have  been  developed,  one  for  fixed  wing 
aircraft  and  one  for  helicopters.  Both  new  trackers  use  orientation  data  in  order  to 
substantially  improve  the  accuracy  of  target  trajectory  predictions.  Specifically, 
an  extended  nonlinear  kalman  filter  has  been  designed  that  uses  radar  position 
and  velocity  information  and  optical  attitude  measurements  of  the  aircraft  being 
tracked. 


IL  Summary  of  the  Most  Important  Results 


A.  Partial  Shape  Recognition 

This  research  effort  involved  recognizing  a  shape  when  part  of  the  boundary  is 
incorrect  due  to  obscuration  or  poor  segmentation.  The  first  method  involves 
segmenting  a  contour  into  pieces  based  on  the  breakpoints  generated  by  a 
polygonal  line  fit,  and  describing  each  segment  with  Fourier  descriptors.  When 
two  contours  are  to  be  compared,  the  distances  between  each  of  their  respective 
segment  sets  may  be  viewed  as  an  inter-segment  distance  table.  Emphasis  was 
placed  on  the  use  of  dynamic  programming  techniques  to  find  a  subset  of  shape 
features  along  a  contour  which  match  a  subset  of  those  in  a  library  shape, 
independent  of  translation,  rotation,  and  scale  of  the  shape.  Special  attention  was 
paid  to  the  effects  of  noise  on  this  technique  and  modifications  which  can  be  made 
to  make  the  technique  less  sensitive  to  noise  and  resolution  changes.  The  dynamic 
programming  approach  developed  for  this  problem  finds  a  path  through  the 
distance  table  which  gives  the  best  local  matches  independent  of  any  few  bad 
matches  along  the  path.  Thus  the  normal  dynamic  programming  approach 
which  results  in  a  global  optimum  path  has  been  modified  to  find  a  large  number 
of  local  optimums  independent  of  whether  it  results  in  a  global  optimum.  Tests 
on  partial  aircraft  contours  and  other  shapes  have  shown  that  this  method  works 
very  well.  The  method  is  described  in  the  Ph.D.  thesis  by  John  Gorman  [2]  and  in 
the  IEEE  Transaction  paper  by  Gorman,  Mitchell,  and  Kuhl  [21]. 

A  demonstration  was  implemented  which  allows  a  TV  camera  to  image  various 
aircraft  scale  models.  The  data  is  digitized  and  aircraft  identification  and 
orientation  estimation  is  performed.  This  can  be  done  even  when  the  aircraft 
models  are  partially  obscured  from  the  view  of  the  TV  camera.  This  method, 
although  accurate,  does  require  that  segmentation  is  done  so  that  a  reasonable 
piece  of  the  contour  is  available  for  analysis  by  this  program.  Some  effort  would  be 
required  to  achieve  the  speed  necessary  for  real  time  operation  of  this  algorithm. 

A  second  approach  to  partial  shape  recognition  was  based  on  curvature 
measurements  of  pieces  of  edges  extracted  from  2-D  images.  The  advantages  of 
this  method  over  the  first  is  that  no  segmentation  step  is  required  and  the 
algorithm  requires  less  computation.  An  edge  operator  is  used  to  find  edge 
segments  (not  necessarily  connected)  that  are  compared  to  libraries  of  shapes. 
This  is  especially  useful  under  camouflage  situations  where  the  true  contour  may 
be  disconnected  at  many  locations.  For  these  contour  pieces  which  nay  be 
disconnected  and  non-ordered,  sets  of  significant  vertices  are  extracted.  The 
locations  of  these  vertices  are  accurately  preserved  by  the  detection  process,  even 
when  the  contours  are  extracted  from  medium  resolution  (50-100  pixel  area) 
objects.  The  method  used  to  extract  these  features  is  scale  invariant.  The  vertex 
features  are  then  rank  ordered  according  to  a  significance  measure  and  used  to 
hypothesize  matches  to  the  library  shapes.  The  ordering  used  significantly 
reduces  the  number  of  combinations  which  must  be  analyzed  to  find  a  match. 

The  matching  process  is  hierarchical  in  that  significant  vertex  matches  are  then 
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followed  by  secondary  vertex  matches  and  finally  by  a  tolerance  band  contour 
match.  A  complete  description  of  the  algorithm  can  be  found  in  the  Ph.D.  thesis 
by  Douglas  Hung  [5]. 


B.  Orientation  from  Generic  Shape  Matching 

Generic  shape  recognition  is  defined  to  be  the  recognition  of  an  object 
as  a  member  of  a  class  of  objects  rather  than  as  a  specific  object  in  a 
class.  For  example  we  may  wish  to  recognize  an  object  as  an  airplane  or  a 
helicopter  without  classifying  as  to  specific  type.  In  addition  to  recognition  of 
category,  it  might  be  very  useful  to  determine  orientation  of  the  airplane  or 
helicopter  for  purposes  of  predicting  its  future  trajectory.  The  advantages  of 
generic  recognition  are  that  no  specific  library  entry  is  required  (that  is  an 
airplane  could  be  recognized  and  tracked,  even  if  it  had  not  been  seen  before)  and 
that  the  recognition  process  has  the  potential  of  being  very  fast,  since  no  library 
comparisons  are  involved,  only  the  matching  of  extracted  features  to  a  set  of  rules 
describing  the  generic  category. 

By  investigating  various  basic  structural  properties  of  many  aircraft  shapes,  we 
have  found  that  the  skeleton  of  the  shape  can  most  effectively  be  used  for  generic 
recognition.  (When  a  human  is  shown  these  skeletons,  recognition  of  aircraft 
category  and  orientation  is  usually  possible).  We  have  developed  a  rule  based 
system  approach  to  generic  aircraft  recognition  which  incorporates  the  following 
steps:  (1)  extract  a  silhouette,  (2)  reduce  the  silhouette  to  a  skeleton  (line  drawing), 
(3)  match  straight-line  segments  to  the  skeleton,  (4)  label  the  line-segments  as 
aircraft  components  according  to  a  set  of  rules,  and  (5)  use  the  length  and  angles 
of  the  labeled  components  to  estimated  aircraft  orientation.  Tests  of  this  system  on 
graphically  generated  aircraft  (of  various  types)  from  random  aspect  angles  have 
shown  that  4%  were  incorrectly  classified,  20%  were  labeled  unrecognizable,  and 
the  remaining  76%  were  correctly  classified  with  average  roll,  pitch,  and  yaw 
errors  of  under  10  degrees.  Details  of  this  technique  and  the  results  are  given  in 
the  Ph.D.  thesis  by  John  Gorman  [2],  Extensions  of  this  method  have  been  made 
to  recognition  of  industrial  parts  for  robot  vision  systems. 


C.  Representation  and  Comparison  of  3-D  Shape  Data 

The  use  of  spherical  harmonics  to  represent  3-D  objects  has  been  investigated. 
Spherical  harmonics  date  back  to  the  eighteenth  century  and  have  been  used  often 
in  the  field  of  Potential  Theory.  We  have  looked  at  the  possibility  of  using  such  a 
representation  to  convert  range  data  to  a  3-D  model  so  that  3-D  shape  comparisons 
can  be  made.  To  this  point  we  have  demonstrated  calculation  of  coefficients  from 
range  data,  reconstruction  from  coefficient  data,  normalization  of  scale, 
translation,  and  rotation,  and  comparison  of  two  shapes  from  their  coefficient 
representation.  This  is  also  a  viable  method  for  matching  shapes  using  range 
map  data,  even  when  the  back  side  of  the  object  is  not  visible  (2  1/2  dimensional 
data).  A  complete  2  1/2  -  D  shape  recognition  algorithm  has  been  developed  and 
implemented  which  involves  the  matching  of  spherical  harmonic  coefficients 


with  library  entries  representing  the  range  data  expected  for  various  viewing 
aspect  angles.  A  method  of  scale  and  viewing  axis  rotation  normalization  has 
been  developed  so  that  shape  matching  can  be  accomplished  independent  of  these 
variables.  Details  of  this  method  can  be  found  in  the  paper  by  Gorman  and 
Mitchell  [17]  and  the  Ph.D.  thesis  by  Gorman  [2]. 


D.  Segmentation 

In  this  area  we  concentrated  on  improving  and  speeding  up  existing 
segmentation  algorithms  which  use  edge  and  texture  data  as  well  as  developing 
new  fast  segmentation  methods  using  model  based  analysis  of  the  gray  level 
histograms  to  select  appropriate  thresholds.  The  purpose  of  this  segmentation 
research  has  been  to  allow  us  to  extract  the  shape  data  from  real  images  so  that 
recognition  and  tracking  can  be  performed. 

A  major  problem  in  histogram  based  image  segmentation  is  that  of  optimum 
threshold  selection.  We  have  examined  the  use  of  median  filtering  techniques  to 
smooth  the  histogram.  An  algorithm  has  been  developed  which  uses  the  root 
signal  resulting  from  successive  median  filtering  to  find  the  major  peaks  of  the 
histogram.  An  empirical  measure  was  developed  to  test  whether  the  histogram 
is  unimodal  or  multimodal.  The  algorithm  is  very  fast  and  provides  results 
comparable  to  the  iterative,  discriminant,  and  entropy  techniques  with  normal 
bimodal  high  contrast  images.  It  is  superior  to  the  other  techniques  with  respect 
to  low  contrast  images  and  allows  a  user  to  adjust  the  performance  by  changing 
the  separability  measure  [9]. 

In  addition  to  threshold  selection  techniques,  a  new  approach  to  adaptive 
thresholding  was  implemented  which  estimates  the  background  statistics 
adaptively  at  each  pixel  based  on  assumptions  that  the  low  variance  blocks  on  the 
border  of  the  image  are  most  likely  background  regions.  The  image  is  then 
segmented  into  regions  which  have  labels:  much  darker  than  background,  darker 
than  background,  uncertain  dark,  background,  uncertain  light,  lighter  than 
background,  and  much  lighter  than  background.  This  technique  which  has 
shown  dramatic  results  is  discussed  completely  in  the  Ph.D.  thesis  by 
Rodriguez  [4],  and  in  the  papers  by  Rodriguez  and  Mitchell  [27],  [28],  [34],  [35]. 


E.  Tracking  of  Maneuvering  Targets  using  Attitude  Measurements 

Video  processing  methods  described  above  provide  estimates  of  an  aircraft's 
orientation  in  space.  The  three  orientation  angles  can  be  used  in  a  target  tracker 
to  improve  tracking  performance.  Advanced  trackers  of  this  type  utilize  a 
mathematical  model  of  the  force  system  on  the  aircraft  that  is  a  function  of  vehicle 
attitude  and  velocity.  The  fundamental  problem  in  target  tracking  is  that  target 
aircraft  do  not  fly  in  straight  lines  during  tactical  situations.  Rather,  they  will 
accelerate  in  a  violent,  time-varying  fashion  (i.e.,  jink)  to  avoid  being  shot  down. 
The  advantage  of  jinking  is  that  the  future  trajectory  of  the  vehicle  becomes  more 


difficult  to  predict,  making  computation  of  the  gun  lead  angle  substantially  less 
accurate.  Lead  times  of  1  to  4  seconds  are  typical  and  during  this  period  an 
aircraft  can  maneuver  away  from  tracker  gun  fire.  In  this  work,  a  new  tracker 
has  been  developed  and  successfully  tested  in  which  the  model  of  vehicle 
acceleration  is  substantially  improved  by  modeling  vehicle  rotation  states  and  by 
incorporating  vehicle  attitude  measurements.  Specifically,  an  extended 
nonlinear  kalman  filter  has  been  designed  that  uses  radar  position  and  velocity 
information  and  optical  attitude  measurements  of  the  aircraft  being  tracked. 
Background  material  on  the  new  tracker  has  been  published  in  the  paper  “A 
Nonlinear  Tracker  Using  Attitude  Information,”  by  Andrisani,  Kuhl,  and 
Gleason  [10]. 

Two  new  classes  of  target  trackers  have  been  developed,  one  for  fixed  wing 
aircraft  and  one  for  helicopters.  Both  new  trackers  used  orientation  data  as  might 
be  provided  by  an  optical  image  processor  in  order  to  substantially  improve  the 
accuracy  of  target  trajectory  predictions.  Orientation  was  found  to  be  useful  in 
tracking  because  when  a  vehicle  is  gr.ng  to  turn,  the  first  thing  that  happens  is 
the  orientation  changes.  This  has  the  effect  of  tilting  the  lift  or  thrust  vector  that 
acts  on  the  vehicle  and  the  vehicle  trajectory  turns  under  the  influence  of  this  new 
force  system.  Because  orientation  always  leads  trajectory  changes,  orientation  is 
useful  in  determining  where  the  \ehicle  is  going  in  the  future;  i.e.,  in  predicting 
future  trajectories.  Predicting  up  to  5  seconds  into  the  future  is  an  exceedingly 
important  part  of  the  fire  control  problem  since  it  typically  takes  that  long  to 
deliver  ordinance  from  the  defender  to  the  attacker. 

Our  approach  was  to  use  orientation  information  to  help  determine  the  direction 
and  magnitude  of  the  lift  or  thrust  vector  and  from  this  determine  the  present  and 
future  trajectory.  In  the  fixed  wing  case  we  used  orientation  of  the  fuselage  (e.g. 
euler  angles)  while  in  the  helicopter  case  we  used  orientation  of  the  tip  path  plane 
formed  by  the  rotor  blades  as  they  turn  above  the  fuselage  in  addition  to  fuselage 
orientation.  It  was  found  that  the  orientation  of  the  tip  path  plane  was  helpful  but 
not  required  in  helicopter  tracking  because  tip  path  angles  are  typically  smau. 
The  orientation  of  the  helicopter  fuselage  was  sufficient  to  produce  dramatic 
improvements  in  trajectory  predictions. 

A  direct  comparison  of  trackers  using  attitude  measurements  and  trackers 
without  attitude  was  initiated.  Tracking  software  which  employs  attitude 
information  was  transported  to  ARDC  in  December  1986.  Modification  to  the 
software  was  done  in  order  to  use  ARDC  flight  data  as  input  and  to  compute 
tracking  accuracy  measures  in  ARDC  format.  Comparisons  with  existing 
trackers  on  ARDC  flight  data  show  that  the  Purdue  tracker  achieves  azimuth  and 
elevation  pointing  errors  3-4  times  smaller  than  conventional  trackers  that  do  not 
use  attitude  angles  or  radar  rate  information. 

Our  results  indicate  that  the  heading  axis  is  a  better  axis  system  in  which  to 
model  target  accelerations  then  the  inertial  axis  system.  Furthermore  it  has  been 
demonstrated  that  many  of  the  nonlinear  relationships  in  the  mathematical 
model  of  the  aircraft  can  be  replaced  by  linear  relationships  without  noticeable 
loss  of  accuracy. 


10 


Work  reported  in  the  paper  “Aircraft  Lead  Angle  Production”  [12]  represents  a 
definitive  comparison  between  trackers  that  use  attitude  and  trackers  that  do  not 
use  attitude.  Overall  the  improvement  in  trajectory  prediction  accuracy  derived  by 
using  attitude  measurements  was  over  300%.  Research  described  in  “Helicopter 
Tracking  Using  Attitude  Information”  [13]  involves  a  first  look  at  using  helicopter 
body  angle  and  rotor  tip  path  plane  angle  measurements  to  aid  in  tracking 
maneuvering  helicopters.  Additional  results  are  given  in  [15].  The  research  is 
continuing  through  other  Army  funding.  The  paper  entitled  “Tracker  Parameter 
Optimization”  [14]  presents  an  analytical  approach  to  oxning  tracking  filters 
when  the  target  is  accelerating  at  various  g  levels. 


F.  Other  Image  Processing  Results 

Adilitional  results  have  been  achieved  which  are  related  to  the  support  provided  by 
thL  contract.  A  significant  effort  has  been  made  in  development  precision 
measurement  tools,  for  edge  location  [6],  [36],  edge  orientation  [11],  [26],  and 
range  [16],  Improvements  in  video  coding  have  resulted  [7].  Mathematical 
morphology  methods  for  image  analysis  have  been  further  developed  [3],  [18],  [19], 
[22],  [23],  [30],  Techniques  for  visual  servoing  and  tracking  have  been 
implemented  [29],  [31],  [32],  [33].  In  addition,  the  techniques  developed  here  have 
been  applied  to  CAD-based  inspection  of  manufactured  ’  arts  [1],  [20],  [24],  [2u]. 
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